Class notes: Monte Carlo methods Week 3, Markov chain Monte Carlo

Markov chain Monte Carlo, or MCMC, is a way to sample probability distributions that cannot be sampled practically using direct samplers. Most complex probability distributions in more than a few variables are are sampled in this way. For us, a stationary Markov chain is a random sequence X1, X2, . . ., where Xk+1 = M(Xk, ξk), where M(x, ξ) is a fixed function and the inputs ξ are i.i.d. random variables. Mathematically, a stationary Markov chain is defined by a transition distribution, R, that describes distribution of Xk+1 conditional on Xk. We use different related notations for this, including R(y|x) for the probability density of Y = M(X, ξ), or Rxy = P(x→ y) for the probability that Xk+1 = y given that Xk = x. An MCMC sampler is a code that implements M . It is a direct sampler for the conditional distribution R(·|Xk). A direct sampler is X = fSamp(), while the MCMC sampler is X = fSamp(X). Both the direct and MCMC samplers can call uSamp() many times. The Markov chain is designed so that fk → f as k →∞, where fk is the distribution of Xk and f is the target distribution. It turns out to be possible to create suitable practical Markov chains for many distributions that do not have practical direct samplers. Two theorems underly the application of MCMC, the Perron Frobenius theorem, and the ergodic theorem for Markov chains. A Markov chain preserves f if f is an invariant distribution for R. This means that if Xk ∼ f , then Xk+1 ∼ f . Perron Frobenius says, among other things, a Markov chain preserves f , and if it is non-degenerate (aperiodic and irreducible, see below), then fk → f as k →∞. It is usually easy to check the non-degeneracy conditions. The importance of the Perron Frobenius theorem is that we do not have to design MCMC samplers that make fk converge to f , we only have to make them preserve f . The MCMC samples Xk are not independent, but they suffice for estimating expected values. The ergodic theorem for Markov chains says that if R preserves f and is non-degenerate then